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CLAIMS 

1- A virtual viewpoint image generation 
method comprising: 
5 a step of obtaining plural images of an 

object taken by plural cameras; 

a step of determining a virtual viewpoint 
that is a position from which the object is viewed; 
and 

10 a step of generating a virtual viewpoint 

image that is an image of the object viewed from the 
virtual viewpoint based on the obtained images of 
the object, 

the step of generating the virtual 
15 viewpoint image comprising: 

a step 1 of setting projection planes 
having a multi-layered structure; 

a step 2 of obtaining each corresponding 
point, on the images of the object, corresponding to 
20 a projection point of a projection plane; 

a step 3 of determining color information 
or brightness information of each projection point 
based on color information or brightness information 
of corresponding points; 
25 a step of calculating, for each of the 

projection points overlapping when viewed from a 
reference viewpoint in a space, a degree of 
probability that the object exists at a distance 
corresponding to a position of the projection point 
30 based on a degree of correlation of the 

corresponding points or neighborhoods of the 
corresponding points; 

a step 5 of performing mixing processing 
on color information or brightness information of 
35 reference points overlapping when viewed from the 
virtual viewpoint according to the degree of 
probability of existence of the object so as to 
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determine color information or brightness 
information of each pixel of the virtual viewpoint 
image; and 

a step 6 of repeating steps from the step 
5 1 to the steps 5 for every point corresponding to 
pixels of the virtual viewpoint image - 



10 

2. The virtual viewpoint image generation 
method as claimed in claim 1, the step 3 comprising: 

mixing the color information or the 
brightness information of the corresponding points 
15 or selecting the color information or the brightness 
information of one corresponding point from the 
color information or the brightness information of 
the corresponding points. 

20 



3. The virtual viewpoint image generation 
method as claimed in claim 1 or 2, the step 4 or the 
25 step 5 comprising: 

a step of setting, in each reference point 
on the projection plane, transparency having plural 
gradations from transparent to opaque by converting 
the degree of probability that the object exists; 
30 and 

the step 5 comprising: 

performing the mixing processing according 
to the transparency instead of the degree of 
probability that the object exists. ^ 

35 
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4. The virtual viewpoint image generation 
method as claimed in claim 3 f the mixing processing 
of the step 5 comprising: 
5 processing projection points successively 

from a projection point far from the virtual 
viewpoint to a projection point near the virtual 
viewpoint , 

wherein color information or brightness 
10 information obtained by the mixing processing up to 
a projection point is obtained by calculating 
interior division between color information or 
brightness information at the projection point and 
color information or brightness information obtained 
15 by the mixing processing up to a previous projection 
point in a ratio according to the transparency - 



5. The virtual viewpoint image generation 
method as claimed in any one of claims 1-4 , wherein: 
projection planes specific to each camera 
taking each image of the object are set in the step 
25 1; 

the color information or the brightness 
information of the step 3 are determined only by 
using color information or brightness information of 
the corresponding points of the images of the object 
30 taken by the plural cameras; 

the degree of probability that the object 
exists in step 4 is calculated using, as the 
reference viewpoint, a viewpoint of the camera 
specific to the projection plane to which the 
35 projection point belongs; and 

correction is performed based on position 
relationship between the virtual viewpoint and each 
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reference viewpoint in the mixing processing of the 
color information or the brightness information. 



6- A virtual viewpoint image generation 
apparatus comprising: 

an object image obtaining means for 
10 obtaining plural images of an object taken by plural 
cameras ; 

a virtual viewpoint determination means 
for- determining a virtual viewpoint, that is a 
position from which the object is viewed; and 

15 an image generation means for generating a 

virtual viewpoint image that is an image of the 
object viewed from the virtual viewpoint based on 
the obtained images of the object, 

the image generation means comprising: 

20 a projection plane determination means for 

determining projection planes having a multi-layered 
structure; 

a reference viewpoint determination means 
for determining a position of the reference 
25 viewpoint; 

a texture array keeping means for keeping 
an array of texture images to be mapped to the 
projection planes; 

a corresponding point matching processing 
30 means for associating parts, in the images of the 
object, on which the same region of the object 
appears with each other; 

a color information determination means 
for determining color information or brightness 
35 information in the array of the texture images by 
performing mixing processing on the images of the 
object; 
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an existence probability information 
determination means for calculating a degree of 
probability that the object exists at a distance 
corresponding to a position of the projection point 
5 in the array of the texture images based on the 
processing result of the corresponding point 
matching processing means; 

a rendering means for rendering the 
projection planes viewed from the virtual viewpoint 
10 based on the color information or the brightness 
information determined by the color information 
determination means and the existence probability 
determined by the existence probability 
determination means, 

15 



7. The virtual viewpoint image generation 
20 apparatus as claimed in claim 6, the existence 
probability information determination means 
comprising : 

a means for setting, in each reference 
point on the projection plane, transparency having 
25 plural gradations from transparent to opaque by 

converting the degree of possibility that the object 
exists ; 

wherein the rendering means performs 
rendering using the transparency instead of the 
30 degree of possibility that the object exists. 



35 8. The virtual viewpoint image generation 

apparatus as claimed in claim 7, the rendering means 
comprising: 
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a means for processing projection points 
successively from a projection point far from the 
virtual viewpoint to a projection point near the 
virtual viewpoint, 
5 wherein color information or brightness 

information obtained by the mixing processing up to 
a projection point is obtained by calculating 
interior division between color information or 
brightness information at the projection point and 
10 color information or brightness information obtained 
by the mixing processing up to a previous projection 
point in a ratio according to the transparency - 



15 

9. The virtual viewpoint image generation 
apparatus as claimed in any one of claims 6-8 , 
wherein: 

20 the projection plane determination means 

determines projection planes specific to each camera 
taking each image of the object; 

the color information determination means 
determines the color information or the brightness 

25 information only by using color information or 

brightness information of the corresponding points 
of the images of the object taken by the plural 
cameras ; 

the existence probability information 
30 determination means calculates the degree of 

probability that the object exists by using, as the 
reference viewpoint, a viewpoint of the camera 
specific to the projection plane to which the 
projection point belongs; and 
35 the rendering means includes a means for 

performing correction based on position relationship 
between the virtual viewpoint and each reference 
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viewpoint . 



5 

10. A virtual viewpoint image generation 
program causing a computer to perform: 

a step of obtaining plural images of an 
object taken by plural cameras; 
10 a step of determining a virtual viewpoint 

that is a position from which the object is viewed; 
and 

a step of generating a virtual viewpoint 
image that is an image of the object viewed from the 
15 virtual viewpoint based on the obtained images of 
the object, 

the step of generating the virtual 
viewpoint image comprising: 

a step 1 of setting projection planes 
20 having a multi-layered structure; 

a step 2 of obtaining each corresponding 
point, on the images of the object, corresponding to 
a projection point of a projection plane; 

a step 3 of determining color information 
25 or brightness information of each projection point 

based on color information or brightness information 
of corresponding points? 

a step 4 of calculating, for each of the 
projection points overlapping when viewed from a 
30 reference viewpoint in a space, a degree of 

probability that the object exists at a distance 
corresponding to a position of the projection point 
based on a degree of correlation of the 
corresponding points or neighborhoods of the 
35 corresponding points; 

a step 5 of performing mixing processing 
on color information or brightness information of 
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reference points overlapping when viewed from the 
virtual viewpoint according to the degree of 
probability of existence of the object so as to 
determine color information or brightness 
5 information of each pixel of the virtual viewpoint 
image; and 

a step 6 of repeating steps from the step 
1 to the steps 5 for every point corresponding to 
pixels of the virtual viewpoint image. 



11. A computer readable recording medium 
15 storing the virtual viewpoint image generation 
program as claimed in claim 10. 



20 

12. An image generation method 
comprising : 

a step of obtaining images of an object 
taken from different viewpoints; 
25 a step of obtaining a three-dimensional 

shape of the object based on the images; and 

a step of generating an image of the 
object viewed from a viewpoint of an observer based 
on the obtained three-dimensional shape of the 
30 object, 

the step of obtaining the three- 
dimensional shape of the object comprising: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
35 dimensional space; 

a step of determining a reference 
viewpoint for obtaining the three-dimensional shape 
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of the object; 

a step of determining color information or 
brightness information of projection points that are 
points on the projection planes based on color 
5 information or brightness information of 

corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of calculating correlation degrees 
among corresponding points corresponding to the 

10 projection points; 

a step of calculating, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that the object exists at the 
15 projection point based on the correlation degree of 
each projection point; 

the step of calculating the correlation 

degree comprising: 

a step of preparing plural groups of 
20 camera sets each being a group of some viewpoints 
selected from the plural viewpoints; and 

a step of obtaining the correlation degree 
from the corresponding points on images included in 

each camera set, 

the step of determining the existence 



25 



30 



probability comprising: 

a step of calculating the existence 
probability based on the correlation degree of each 
projection point obtained for each camera set; and 

a step of determining the existence 
probability of each projection point by performing 
integrating processing for the existence probabUxty 
determined for each camera set. 



35 
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13. The image display method as claimed 
in ciaim 12, the step of calculating the existence 
probability based on the correlation degree of each 
projection point obtained for each camera set 

5 comprising: 

a step of calculating an evaluation 
reference value based on the correlation degreee of 
each projection point calculated for each camera 

S © t r 

10 a step of calculating a distribution 

function of the existence probability by performing 
statistical processing on the evaluation reference 
value of each projection point calculated for each 

camera set; and 
15 a step of determining the existence 

probability of each projection point based on the 
distribution function of the existence probability. 



20 

14. The image generation method as claimed 
in claim 12 or 13, the step of generating the image 
of the object viewed from the viewpoint of the 

25 observer comprising: 

mixing color information or brightness 
information of the projection points overlapping 
when viewed from the viewpoint of the observer in a 
ratio of the existence probability to determine 

30 color information or brightness information of each 
point on the image to be generated so as to generate 
a two-dimensional image. 



35 



15. The image generation method 
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claimed in claim 12 or 13, the step of generating 
the image of the object viewed from the viewpoint of 
the observer comprising: 

a step of setting plural image 
generation planes at positions having different 
depths when viewed from the viewpoint of the 

observer; and 

a step of converting the color 
information or brightness information and the 
existence probability of each projection point into 
color information or brightness information and a 
brightness distribution coefficient on each image 
generation plane based on position relationship 
among projection points overlapping when viewed from 
the viewpoint of the observer and points on each 
image generation plane. 



20 

16. An image generation apparatus 

comprising: 

an object image obtaining means for 
obtaining images of an object taken from different 

25 viewpoints; 

a three-dimensional shape obtaining means 

for obtaining a three-dimensional shape of the 
object based on the images; and 

an object image generation means for 
30 generating an image of the object viewed from a 
viewpoint of an observer based on the obtained 
three-dimensional shape of the object, 

the three-dimensional shape obtaining 

means comprising: 
35 a means for setting projection planes 

having a multi-layered structure in a virtual three- 
dimensional space; 
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a means for determining a reference 
viewpoint for obtaining the three-dimensional shape 

of the object; 

a means for determining color information 

5 or brightness information of projection points that 

are points on the projection planes based on color 

information or brightness information of 

corresponding points, on the obtained images, 

corresponding to the projection points ; 

10 a means for calculating correlation 

degrees among corresponding points corresponding to 

the projection points ; 

a means for determining, for each of the 
projection points overlapping when viewed from the 
15 reference viewpoint, an existence probability that 
is a probability that the object exists at the 
projection point based on the correlation degree of 

each projection point; 

the means for calculating the correlation 

20 degree comprising: 

a means for preparing plural groups of 
camera sets each being a group of some viewpoints 
selected from the plural viewpoints; and 

a means for obtaining the correlation 
25 degree from the corresponding points on images 
included in each camera set, 

the means for determining the existence 

probability comprising: 

a means for calculating the existence 



30 



probability based on the correlation degree of each 
projection point obtained for each camera set; and 

a means for determining the existence 
probability of each projection point by P~*°»£« 
integrating processing for the existence probability 
35 determined for each camera set. 
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17. The image generation apparatus as 
claimed in claim 16, the means for calculating the 
5 existence probability based on the correlation 

degree of each projection point obtained for each 
camera set comprising: 

a means for calculating an evaluation 
reference value based on the correlation value of 
10 each projection point calculated for each camera 
set; 

a means for calculating a distribution 
function of the existence probability by performing 
statistical processing on the evaluation reference 
15 value of each projection point calculated for each 

camera set; and 

a means for determining the existence 
probability of each projection point based on the 
distribution function of the existence probability. 

20 



25 



30 



16. The image generation apparatus as 
claimed in claim 16 or 17, wherein the means for 
generating the image of the object viewed from the 
viewpoint of the observer is a means for: 

mixing color information or brightness 
information of the projection points overlapping 
when viewed from the viewpoint of the observer m a 
ratio of the existence probability to determine 
color information or brightness information of each 
point on the image to be generated so as to 
determine a two-dimensional image. 



35 
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19. The image generation apparatus as 
claimed in claim 16 or 17, the means for generating 
the image of the object viewed from the viewpoint of 
5 the observer comprising: 

a means for setting plural image 
generation planes at positions having different 
depths when viewed from the viewpoint of the 
observer; and 

10 a means for converting the color 

information or brightness information and the 
existence probability of each projection point into 
color information or brightness information and a 
brightness distribution coefficient on each image 

15 generation plane based on position relationship 

among projection points overlapping viewed from the 
viewpoint of the observer and points on the image 
generation planes. 

20 



20. An image generation program causing a 
computer to perform: 
25 a step of obtaining images of an object 

taken from different viewpoints; 

a step of obtaining a three-dimensional 
shape of the object based on the images; and 

a step of generating an image of the 
30 object viewed from a viewpoint of an observer based 
on the obtained three-dimensional shape of the 
object, 

the step of obtaining the three- 
dimensional shape of the object comprising: 
35 a step of setting projection planes having 

a multi-layered structure in a virtual three- 
dimensional space; 
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a step of determining a reference 
viewpoint for obtaining the three-dimensional shape 

of the object; 

a step of determining color information or 
5 brightness information of projection points that are 
points on the projection planes based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 
10 a step of calculating correlation degrees 

among corresponding points corresponding to the 

projection points; 

a step of calculating, for each of the 
projection points overlapping when viewed from the 
15 reference viewpoint, an existence probability that 
is a probability that the object exists at the 
projection point based on the correlation degree of 
each projection point; 

the step of calculating the correlation 

20 degree comprising: 

a step of preparing plural groups of 
camera sets each being a group of some viewpoints 
selected from the plural viewpoints; and 

a step of obtaining the correlation degree 
from the corresponding points on images included in 



25 



30 



each camera set, 

the step of determining the existence 



probability comprising: 

a step of calculating the existence 
probability based on the correlation degree of each 
projection point obtained for each camera set; and 

a g t ep of determining the existence 
probability of each projection point by performing 
integrating processing for the existence probability 
35 determined for each camera set. 
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21. A computer readable recording medium 
storing the image generation program as claimed in 
5 claim 20. 



10 22. An image generation method comprising: 

a step of obtaining images of an object 
taken by changing focusing distance; 

a step of obtaining a virtual viewpoint 
that is a viewpoint from which the object appearing 
15 in the images is viewed; 

a step of obtaining a three-dimensional 
shape of the object based on the images; and 

a step of generating an image of the 
object viewed from the virtual viewpoint based on 
20 the obtained three-dimensional shape of the object, 
the step of obtaining the three- 
dimensional shape of the object comprising: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
25 dimensional space; 

a step of determining a reference 
viewpoint for obtaining the three-dimensional shape 

of the object; 

a step of determining color information or 

30 brightness information of projection points, that 

are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

35 a step of determining focusing degrees of 

the projection points based on focusing degrees of 
the corresponding points corresponding to the 
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projection points ; 

a step of calculating, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
5 is a probability that the object exists at a 
distance corresponding to a position of the 
projection point based on the focusing degree of 
each projection point; 

the step of generating the image of the 
10 object viewed from the virtual viewpoint comprising: 
mixing color information or brightness 
information of the projection points overlapping 
when viewed from the virtual viewpoint in a ratio 
corresponding to the existence probability to 
15 determine color information or brightness 

information of each point of the image to be 
generated. 



20 



25 



23. The image generation method as claimed 
in claim 22, the step of obtaining the three- 
dimensional shape of the object or the step of 
generating the image of the object viewed from the 
virtual viewpoint comprising: 

a step of setting transparency having 
plural gradations from transparent to opaque on each 
projection point based on the existence probability 
of the projection points overlapping when viewed 
from the reference viewpoint or the virtual 
viewpoint; 

the step of generating the image of the 
object viewed from the virtual viewpoint comprising: 
35 mixing the color information or the 

brightness information of the projection points 
overlapping when viewed from the virtual viewpoint 



30 



2005$12fll6B 2 28§35# I TOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 249 



-239- 

in a ratio according to the transparency set based 
on the existence probability so as to determine the 
color information or brightness information of each 
point of the image to be generated. 

5 



24. The image generation method as claimed 

10 in claim 23, 

the step of generating the image of the 
object viewed from the virtual viewpoint comprising: 

mixing the color information or the 
brightness information for projection points 
15 successively from a projection point far from the 
virtual viewpoint to a projection point near the 
virtual viewpoint, 

wherein color information or brightness 
information obtained by the mixing processing up to 
20 a projection point is obtained by calculating 
interior division between color information or 
brightness information at the projection point and 
color information or brightness information obtained 
by the mixing processing up to a previous projection 
25 point in a ratio according to the transparency. 



30 25. An image generation apparatus 

comprising ; 

an object image obtaining means for 
obtaining images of an object taken by changing 
focusing distance; 
35 a virtual viewpoint setting means for 

setting a virtual viewpoint that is a viewpoint from 
which the object appearing in the images is viewed; 
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a three-dimensional shape obtaining means 
for obtaining a three-dimensional shape of the 
object based on the images; and 

a rendering means for generating an image 
5 of the object viewed from the virtual viewpoint 

based on the obtained three-dimensional shape of the 
object, 

the three-dimensional shape obtaining 

means comprising: 
!0 a means for setting projection planes 

having a multi-layered structure in a virtual three- 
dimensional space; 

a means for determining a reference 
viewpoint for obtaining the three-dimensional shape 

15 of the object; 

a means for determining color information 
or brightness information of projection points, that 
are points on the projection planes, based on color 
information or brightness information of 

20 corresponding points, on the obtained images, 
corresponding to the projection points ; 

a means for determining focusing degrees 
of the projection points based on focusing degrees 
of the corresponding points corresponding to the 

25 projection points; 

a means for determining, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that the object exists at a 
30 distance corresponding to a position of the 

projection point based on the focusing degree of 
each projection point; 

the rendering means comprising: 
a means for mixing color information or 
35 brightness information of the projection points 

overlapping when viewed from the virtual viewpoint 
in a ratio corresponding to the existence 
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probability to determine color information or 
brightness information of each point of the image 
be generated. 



26. The image generation apparatus as 
claimed in claim 25, the three-dimensional shape 
10 obtaining means or the rendering means comprising: 
a means for setting transparency having 
plural gradations from transparent to opaque on each 
projection point based on the existence probability 
of the projection points overlapping when viewed 
15 from the reference viewpoint or the virtual 
viewpoint; and 

the rendering means comprising: 
a means for mixing the color information 
or the brightness information of the projection 
20 points overlapping when viewed from the virtual 

viewpoint in a ratio according to the transparency 
set based on the existence probability so as to 
determine the color information or brightness 
information of each point of the image to be 
25 generated. 



30 27. The image generation apparatus as 

claimed in claim 26 r 

the rendering means comprising: 
a means for mixing the color information 
or the brightness information for projection points 
35 successively from a projection point far from the 
virtual viewpoint to a projection point near the 
virtual viewpoint, 
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wherein color information or brightness 
information obtained by the mixing processing up to 
a projection point is obtained by calculating 
interior division between color information or 
5 brightness information at the projection point and 
color information or brightness information obtained 
by the mixing processing up to a previous projection 
point in a ratio according to the transparency. 



28. An image generation program for 
causing a computer to perform: : 
15 a step of obtaining images of an object 

taken by changing focusing lengths; 

a step of obtaining a virtual viewpoint 
that is a viewpoint from which the object appearing 
in the images is viewed; 
20 a step of obtaining a three-dimensional 

shape of the object based on the images; and 

a step of generating an image of the 
object viewed from the virtual viewpoint based on 
the obtained three-dimensional shape of the object, 
25 the step of obtaining the three- 

dimensional shape of the object comprising: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
dimensional space; 
30 a step of determining a reference 

viewpoint for obtaining the three-dimensional shape 

of the object; 

a step of determining color information or 
brightness information of projection points, that 
35 are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
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corresponding to the projection points; 

a step of determining focusing degrees of 
the projection points based on focusing degrees of 
the corresponding points corresponding to the 

5 projection points; 

a step of calculating, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that the object exists at a 
10 distance corresponding to a position of the 

projection point based on the focusing degree of 
each projection point; 

the step of generating the image of the 
object viewed from the virtual viewpoint comprising: 
15 J mixing color information or brightness 

information of the projection points overlapping 
when viewed from the virtual viewpoint in a ratio 
corresponding to the existence probability to 
determine color information or brightness 
20 information of each point of the image to be 
generated . 



25 

29- A computer readable recording medium 
storing the image generation program as claimed in 
claim 28. 



30 



' 30. An image generation method comprising: 
a step of obtaining images of an object 
35 taken under different conditions ; 

a step of obtaining a three-dimensional 
shape of the object based on the images; and 
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a step of generating an image of the 
object viewed from a viewpoint of an observer based 
on the obtained three-dimensional shape of the 
object, 

5 the step of obtaining the three- 

dimensional shape of the object comprising: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
dimensional space; 
10 a step of determining a reference 

viewpoint for obtaining the three-dimensional shape 
of the object ; 

a step of determining color information or 
brightness information of projection points, that 
15 are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining, for each of the 
20 projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 
at the projection point; 

the step of determining the existence 
25 probability comprising: 

a step of calculating an evaluation 
reference value of each projection point from image 
information of the corresponding points; 

a step of performing statistical 
30 processing on the evaluation reference value of each 
projection point; and 

a step of calculating the existence 
probability of each projection point based on the 
evaluation reference value on which the statistical 
35 processing has been performed- 
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31 . The image generation method as 
claimed in claim 30, the step of obtaining the 
5 plural images comprising: 

obtaining images by taking the object from 

different viewpoints; 

the step of determining the existence 

probability comprising: 
10 a step of obtaining correlation degrees 

among corresponding points corresponding to the 

projection points; 

a step of calculating the evaluation 
reference value based on the correlation degree of 
15 each projection point; 

a step of performing the statistical 
processing on the evaluation reference value; and 

a step of calculating the existence 
probability of each projection point based on the 
20 evaluation reference value on which the statistical 
processing has been performed. 



25 

32. The image generation method as claimed 
in claim 30, the step of obtaining the plural images 

comprising: 

obtaining the images of the object taken 

30 from a viewpoint by changing focusing distance; 

the step of determining the existence 

probability comprising: 

a step of calculating focusing degrees of 
the projection points from focusing degrees of the 
35 corresponding points corresponding to the projection 
points; 

a step of calculating the evaluation 
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reference value based on the focusing degree of each 

projection point; 

a step of performing the statistical 
processing on the evaluation reference value; and 

a step of calculating the existence 
probability of each projection point based on the 
evaluation reference value on which the statistical 
processing has been performed. 



10 



20 



33. The image generation method as 
claimed in claim 30, the step of obtaining the 
15 plural images comprising: 

obtaining images of the object taken 
from plural viewpoints, and images of the object 
taken from equal to one or more viewpoints among the 
plural viewpoints by changing focusing distance; 

the step of determining the existence 

probability comprising: 

a step of obtaining correlation degrees 
between the projection points and corresponding 
points on the plural images of different viewpoints; 

a step of obtaining a first evaluation 
reference value based on the correlation degrees of 
each projection point; and 

a step of performing the statistical 

f<T-«i- evaluation reference value; 
processing on the first, evaxuanu 

a step of calculating focusing degrees 
of the projection points based on focusing degrees 
of the corresponding points on the images of 
different focusing distances ; 

a step of calculating a second 
evaluation reference value based on the focusing 
degrees of each projection point; and 

a s tep of calculating the existence 
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probability of each projection point based on the 
first evaluation reference value and the second 
evaluation reference value on which the statistical 
processing has been performed. 

5 



10 



15 



34. The image generation method as 
claimed in any one of claims 30-33, the step of 
generating the image of the object viewed from the 
viewpoint of the observer comprising: 

mixing color information or brightness 
information of the projection points overlapping 
when viewed from the viewpoint of the observer in a 
ratio of the existence probability to determine 
color information or brightness information of each 
point on the image to be generated so as to 
determine a two-dimensional image. 



20 



25 



30 



35 



35 The image generation method as 
claimed in any one of claims 30-33, the step of 
generating the image of the object viewed from the 
viewpoint of the observer comprising: 

a step of setting plural image 
generation planes at positions having different 
depths when viewed from the viewpoint of the . 

observer; and 

a step of converting the color 
information or brightness information and the 
existence probability of each projection point int 
color information or brightness information and a 
brightness distribution coefficient on each image 
generation plane based on position relationship 
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among projection points overlapping viewed from the 
viewpoint of the observer and points on the image 
generation planes. 



10 



20 



36. An image generation apparatus 

comprising: 

an object image obtaining means tor 
obtaining images of an object taken under different 

conditions; 

an object shape obtaining means for 
obtaining a three-dimensional shape of the object 

15 based on the images; and 

an object image generation means for 
generating an image of the object viewed from a 
viewpoint of an observer based on the obtained 
three-dimensional shape of the object, 

the object shape obtaining means 

comprising : 

a means for setting projection planes 

having a multi-layered structure in a virtual three- 

dimensional space; 

a means for determining a reference 
viewpoint for obtaining the three-dimensional shape 

of the object; 

a means for determining color information 
or brightness information of projection points, that 
are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

a means for determining, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 



25 



30 



35 
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at the projection point ; 

the means for determining the existence 

probability comprising: 

a means for calculating an evaluation 
reference value of each projection point from image 
information of the corresponding points; 

a means for performing statistical 
processing on the evaluation reference value of each 

projection point; and 

a means for calculating the existence 
probability of each projection point based on the 
evaluation reference value on which the statistical 
processing has been performed. 



15 



37. The image generation apparatus as 
claimed in claim 36, wherein the object image 
20 obtaining means obtains images of the object taken 
from different viewpoints; 

the means for determining the existence 

probability comprising: 

a means for obtaining correlation degrees 
25 among corresponding points corresponding to the 

projection points; 

a means for calculating the evaluation 

reference value based on the correlation degree of 

each projection point; 
30 a means for performing the statistical 

processing on the evaluation reference value; and 
a means for calculating the existence 

probability of each projection point based on the 

evaluation reference value on which the statistical 
35 processing has been performed. 
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38. The image generation apparatus as 
claimed in claim 36, wherein the object image 
obtaining means obtains the images of the ob 3 ect 
taken from a viewpoint by changing focusing 
distance; 

the means for determining the existence 

probability comprising: 

a means for calculating focusing degrees 
of the projection points from focusing degrees of 
the corresponding points corresponding to the 

projection points; 

a means for calculating the evaluation 
reference value based on the focusing degree of each 

projection point; 

a means for performing the statistical 
processing on the evaluation reference value; and 

a means for calculating the existence 
probability of each projection point based on the 
evaluation reference value on which the statistical 
processing has been performed. 



25 



30 



35 



39. The image generation apparatus as 
claimed in claim 36, wherein the object image 
obtaining means obtains images of the object taken 
from plural viewpoints, and images of the object 
taken from equal to one or more viewpoints among the 
plural viewpoints by changing focusing distance; 

the means for determining the existence 

probability comprising: 

a means for obtaining correlation 
degrees between the projection points and 
corresponding points on the plural images having 
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different projection points; 

a means for obtaining a first evaluation 
reference value based on the correlation degrees of 

each projection point; and 

a means for performing the statistical 
processing on the first evaluation reference value; 

a means for calculating focusing degrees 
of the projection points based on focusing degrees 
of the corresponding points on the images of 
different focusing distances taken from a viewpoint; 

a means for calculating a second 
evaluation reference value based on the focusing 
degrees of each projection point; and 

a means for calculating the existence 
15 probability of each projection point based on the 
first evaluation reference value and the second 
evaluation reference value on which the statistical 
processing has been performed. 



10 



20 



40. The image generation apparatus as 
claimed in any one of claims 36-39, the means for 
25 generating the image of the object viewed from the 
viewpoint of the observer comprising: 

a means for mixing color information or 
brightness information of the projection points 
overlapping when viewed from the viewpoint of the 
observer in a ratio of the existence probability to 
determine color information or brightness 
information of each point on the image to be 
generated so as to determine a two-dimensional image. 



30 



35 
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41. The image generation apparatus as 
claimed in any one of claims 36-39, the means for 
generating the image of the object viewed from the 
viewpoint of the observer comprising: 
5 a means for setting plural image 

generation planes at positions having different 
depths when viewed from the viewpoint of the 

observer; and 

a means for converting the color 

10 information or brightness information and the 

existence probability of each projection point into 
color information or brightness information and a 
brightness distribution coefficient on each image 
generation plane based on position relationship 

15 among projection points overlapping viewed from the 
viewpoint of the observer and points on the image 
generation planes. 



20 



42. An image generation program causing a 

computer to perform: 

a step of obtaining images of an object 

2 5 taken under different conditions; 

a step of obtaining a three-dimensional 
shape of the object based on the images; and 

a step of generating an image of the 
object viewed from a viewpoint of an observer based 
on the obtained three-dimensional shape of the 
object, 

the step of obtaining the three- 
dimensional shape of the object comprising: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
dimensional space; 

a step of determining a reference 
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viewpoint for obtaining the three-dimensional shape 

of the object; 

a step of determining color information or 

brightness information of projection points, that 
5 are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining, for each of the 
10 projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 
at the projection point; 

the step of determining the existence 

15 probability comprising: 

a step of calculating an evaluation 
reference value of each projection point from image 
information of the corresponding points ; 

a step of performing statistical 
20 processing on the evaluation reference value of each 

projection point; and 

a step of calculating the existence 
probability of each projection point based on the 
evaluation reference value on which the statistical 
25 processing has been performed. 



30 43. A computer readable recording medium 

storing the image generation program as claimed in 
claim 42. 



35 

44. A three-dimensional image display 
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method comprising: 

a step of obtaining images of an object 
taken under different conditions ; 

a step of obtaining a three-dimensional 
shape of the object based on the images ; 

a step of setting a viewpoint position 
from which an observer observes plural image display 
planes existing at different depth positions when 
viewed from the observer; 

a step of generating two-dimensional 
images to be displayed on each image display plane 
based on the obtained three-dimensional shape of the 

object; and 

a step of presenting a three-dimensional 

15 image of the object by displaying the generated two- 
dimensional images on each display plane, 

the step of obtaining the three- 
dimensional shape of the object comprising: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
dimensional space; 

a step of determining a reference 
viewpoint for obtaining the three-dimensional shape 

of the object; 

a step of determining color information or 
brightness information of projection points, that 
are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 

35 at the projection point; 

the step of generating the two-dimensional 

images comprising: 
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converting the color information or the 
brightness information and the existence probability 
of the projection point into color information or 
brightness information and existence probability of 
each display point that is a point on the image 
display plane corresponding to the projection plane 
on which the projection point exists so as to 
generate the two dimensional images; 

the step of presenting the three- 
dimensional image of the object comprising: 

displaying the color information or 
brightness information on each display point with 
brightness according to the existence probability. 



15 



45. The three-dimensional image display 
method as claimed in claim 44, the step of obtaining 
20 the plural images comprising: 

obtaining images of the object taken from 

different viewpoints, 

the step of determining the existence 

probability comprising: 
25 a step of calculating correlation degrees 

among corresponding points corresponding to the 

projection points; 

a step of determining, for projection 
points overlapping when viewed from the reference 
30 viewpoint, the existence probability of each 

projection point based on the correlation degree of 
each projection point. 



35 



46. The three-dimensional image display 
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method as claimed in claim 44, the step of obtaining 
the plural images comprising: 

obtaining images of the object taken from 
a viewpoint by changing focusing distance, 
5 the step of determining the existence 

probability comprising: 

a step of calculating focusing degrees of 
the projection points from focusing degrees of 
corresponding points corresponding to the projection 
10 points; 

a step of determining, for projection 
points overlapping when viewed from the reference 
viewpoint, the existence probability of each 
projection point based on the focusing degree of 
15 each projection point. 



20 



47. The three-dimensional image display 
method as claimed in claim 44, the step of obtaining 
the plural images comprising: 

obtaining images of the object taken from 
plural viewpoints, and images of the object taken 
25 from equal to one or more viewpoints among the 

plural viewpoints by changing focusing distance; 

the step of determining the existence 

probability comprising: 

a step of obtaining correlation degrees 
between the projection points and corresponding 
points on the plural images of different viewpoints; 

a step of calculating focusing degrees of 
the projection points based on focusing degrees of 
the corresponding points on the images, having 
35 different focusing distance, of each viewpoint; and 
a step of determining, for projection 
points overlapping when viewed from the reference 
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viewpoints, the existence probability of each 
projection point based on the correlation degree and 
the focusing degree of each projection point. 



48. A three-dimensional image display 

apparatus comprising: 
10 an object image obtaining means for 

obtaining images of an object taken under different 

conditions; 

a three-dimensional shape obtaining means 

for obtaining a three-dimensional shape of the 

15 object based on the images; 

an observer viewpoint setting means for 
setting a viewpoint position from which an observer 
observes plural image display planes existing at 
different depth positions when viewed from the 

20 observer; 

a two-dimensional image generation means 
for generating two-dimensional images to be 
displayed on each image display plane based on the 
obtained three-dimensional shape of the object; and 

wherein the three-dimensional image 
display apparatus presents a three-dimensional image 
of the object by displaying the generated two- 
dimensional images on each display plane, 

the three-dimensional shape obtaining 

30 means comprising: 

. a means for setting projection planes 
having a multi-layered structure in a virtual three- 

dimensional space; 

a means for determining a reference 
35 viewpoint for obtaining the three-dimensional shape 

of the object; 

a means for determining color information 
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or brightness information of projection points, that 
are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
5 corresponding to the projection points; 

a means for determining, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 

10 at the projection point; 

the two-dimensional image generation means 

comprising: 

a means for converting the color 
information or the brightness information and the 
15 existence probability of the projection point into 
color information or brightness information and 
existence probability of each display point that is 
a point on the image display plane corresponding to 
the projection plane on which the projection point 
2 0 exists to generate the two dimensional images; 

wherein the three-dimensional image 
display apparatus displays the color information or 
brightness information on each display point with 
brightness according to the existence probability. 



25 



30 



49. The three-dimensional image display 
apparatus as claimed in claim 48, wherein the object 
image obtaining means is a means for obtaining 
images of the object taken from different viewpoints, 
the means for determining the existence 

probability comprising: 
35 ^ a means for calculating correlation 

degrees among corresponding points corresponding to 
the projection points; 
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a means for determining, for projection 
points overlapping when viewed from the reference 
viewpoint, the existence probability of each 
projection point based on the correlation degree of 
each projection point. 



10 



50. The three-dimensional image display 
apparatus as claimed in claim 48, wherein the object 
image obtaining means is a means for obtaining 
images of the object taken from a viewpoint by 
changing focusing distance, 
15 ~ the means for determining the existence 

probability comprising: 

a means for calculating focusing degrees 
of the projection points from focusing degrees of 
corresponding points corresponding to the projection 

20 points; and 

a means for determining, for projection 
points overlapping when viewed from the reference 
viewpoint, the existence probability of each 
projection point based on the focusing degree of 

25 each projection point. 



30 



51. The three-dimensional image display 
apparatus as claimed in claim 48, wherein the object 
image obtaining means is a means for obtaining 
images of the object taken from plural viewpoints, 
and images of the object taken from equal to one or 
35 more viewpoints among the plural viewpoints by 
changing focusing distance; 

the means for determining the existence 
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probability comprising: 

a means for obtaining correlation 
degrees between the projection points and 
corresponding points on the plural images of 
5 different viewpoints; 

a means for calculating focusing degrees 
of the projection points based on focusing degrees 
of the corresponding points on the images, having 
different focusing length, of each viewpoint; and 
10 a means for determining, for projection 

points overlapping when viewed from the reference 
viewpoints, the existence probability of each 
projection point based on the correlation degree and 
the focusing degree of each projection point. 



52. A three-dimensional image display 
20 program causing a computer to perform: 

a step of obtaining images of an object 
taken under different conditions; 

a step of obtaining a three-dimensional 
shape of the object based on the images; 
25 * a step of setting a viewpoint position 

from which an observer observes plural image display 
planes existing at different depth positions when 
viewed from the observer; 

a step of generating two-dimensional 
30 images to be displayed on each image display plane 

based on the obtained three-dimensional shape of the 

object; and 

a step of presenting a three-dimensional 

image of the object by displaying the generated two- 
35 dimensional images on each display plane, 
the step of obtaining the three- 
dimensional shape of the object comprising: 
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a step of setting projection planes having 
a multi-layered structure in a virtual three- 
dimensional space; 

a step of determining a reference 
5 viewpoint for obtaining the three-dimensional shape 
of the object; 

a step of determining color information or 
brightness information of projection points, that 
are points on the projection planes, based on color 
10 information or brightness information of 

corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining, for each of the 
projection points overlapping when viewed from the 
15 reference viewpoint, an existence probability that 

is a probability that a surface of the object exists 
at the projection point; 

the step of generating the two-dimensional 

images comprising: 

20 a step of converting the color information 

or the brightness information and the existence 
probability of the projection point into color 
information or brightness information and existence 
probability of each display point that is a point on 

25 the image display plane corresponding to the 

projection plane on which the projection point 
exists to generate the two dimensional images; 

the step of presenting the three- 
dimensional image of the object comprising: 

30 a step of displaying the color information 

or brightness information on each display point with 
brightness according to the existence probability. 



35 



53. A computer readable recording medium 
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storing the three-dimensional image generation 
program as claimed in claim 52. 
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